Inserts, Deletes, and Updates that update indexed columns must all maintain indexes as they execute. This is particularly inefficient as every row processed will cause a separate index scan, which could read several blocks. Direct Path SQL*Loader mitigates this effect by deferring the index rebuild until the end of the INSERT.
Depending on the proportion of the table inserted / updated / deleted, it may be faster to drop the indexes (or disable PK and Unique constraints) and then rebuild them after the DML is complete.